Camera using preview image to select exposure

ABSTRACT

A digital camera comprises an image sensor for providing initial sensor image data and final sensor image data; a lens for exposing the image of a scene onto the image sensor; an exposure control system for adjusting an exposure level of a final image on the image sensor in response to a scene type; and a processor for processing the initial sensor image data to select one of a plurality of scene types, and to process the final sensor image data in response to the scene type.

CROSS-REFERENCE TO RELATED PATENT APPLICATIONS

This application is a continuation of U.S. application Ser. No.13/798,692, filed Mar. 13, 2013, which is a continuation of U.S.application Ser. No. 12/265,793, filed Nov. 6, 2008, now U.S. Pat. No.8,488,015, which claims the benefit of U.S. Provisional Appl. No.61/014,852, filed Dec. 19, 2007 and Japanese Appl. No. 2007-335469,filed Dec. 27, 2007, each of which is incorporated by reference in itsentirety.

FIELD

The present disclosure relates to digital cameras that capture digitalimages and, more particularly, to digital cameras that identify the typeof scene to be captured to adjust exposure.

BACKGROUND

Digital cameras are used by a growing number of consumer andprofessional photographers. These cameras use one or more CCD or CMOSimage sensors to capture images, and digitally process these capturedimages to produce digital image files, which are stored in a digitalmemory in the camera.

Prior art Kodak digital cameras use “smart scene” modes to automaticallyidentify a type of scene (e.g. sports, portrait) and to thenautomatically select an appropriate exposure program for the identifiedscene type. The exposure program normally sets the f/number, exposuretime, and ISO speed used when the image is taken.

It is known that the image processing used in a digital camera canadaptively adjust the signal values according to the dynamic range ofthe input image. For example, WO2006018658 “Image Processing Method andComputer Software for Image Processing” assigned to Apical, Ltd.,teaches image correction processing which comprises altering area imageintensity values of an image according to a dynamic range compressionimage transform. While the patent teaches that this image processing canbe used in a digital camera, there is no suggestion that the exposurelevel of the image sensor in the camera be set based on the type ofscene being captured.

It is known that a digital camera can provide image processing toimprove images that appear to be underexposed due to difficult lightingsituations or insufficient flash. For example, recent Nikon CoolPixcameras include a “D-Lighting” function to fix problems that occur withexcessive backlighting and underexposed images. D-Lighting, selected bythe user during playback mode, automatically modifies the image tocompensate for insufficient flash or excessive back lighting. The camerasaves the original image and the corrected copy to the camera's internalor removable memory.

It is known that a digital camera can capture images using a reducedexposure level (e.g. a high ISO setting) in order to preserve more ofthe image highlights. For example, some Canon DSLR cameras include a“highlight tone priority” (HTP) setting, which allows the camera toutilize the much greater headroom available in the sensor pixels whenshooting at elevated ISO settings to recover highlight detail that wouldotherwise be lost.

It is known that digital images, including digital images captured by adigital camera, can be digitally processed to adjust neutral densitybalance and color balance. In particular, adaptive neutral densitybalance adjustment processing can be performed, as described in commonlyassigned U.S. Pat. No. 6,243,133 titled “Method for Automatic SceneBalance of Digital Images” to Spaulding, Gindele and Niederbaumer, thedisclosure of which is incorporated herein by reference. Automatic colorbalance can be performed, as described in commonly assigned U.S. Pat.No. 6,573,932 titled “Method for Automatic White Balance of DigitalImages” to Adams, Hamilton, Gindele and Pillman, the disclosure of whichis incorporated herein by reference. These examples are not limiting,and many other neutral density and color balance adjustment processingsolutions may be used.

It is known that digital images, including digital images captured by adigital camera, can be digitally processed to compensate for thepresence of flare light. In particular, flare compensation processingcan be performed, as described in commonly assigned U.S. Pat. No.6,912,321 titled “Method of Compensating a Digital Image for the Effectsof Flare Light” to Gindele, the disclosure of which is incorporatedherein by reference. This example is not limiting, and many other flarecompensation processing solutions may be used.

It is known that digital images, including digital images captured by adigital camera, can be digitally processed to compensate for the dynamicrange of the scene. In particular, adaptive tone scale adjustmentprocessing can be performed, as described in commonly assigned U.S. Pat.No. 6,937,775 titled “Method of Enhancing the Tone Scale of a DigitalImage to Extend the Linear Response Range Without Amplifying Noise” toGindele and Gallagher, U.S. Pat. No. 7,113,649 titled “Enhancing theTonal Characteristics of Digital Images” to Gindele, U.S. Pat. No.7,130,485 titled “Enhancing the Tonal and Color Characteristics ofDigital Images Using Expansive and Compressive Tone Scale Functions” toGindele and Gallagher, U.S. Pat. No. 7,058,234 titled “Enhancing theTonal, Spatial, and Color Characteristics of Digital Images UsingExpansive and Compressive Tone Scale Functions” to Gindele andGallagher, and U.S. Pat. No. 7,043,090 titled “Enhancing the TonalCharacteristics of Digital Images Using Expansive and Compressive ToneScale Functions” to Gindele and Gallagher, and commonly assigned U.S.Patent Publication No. US20040096103, filed on Nov. 14, 2002 titled“Method of Spatially Filtering a Digital Image Using ChrominanceInformation” to Gallagher and Gindele, and U.S. Patent Publication No.US20040057632, filed on Sep. 19, 2002 titled “Enhancing the TonalCharacteristics of Digital Images Using Inflection Points in a ToneScale Function” to Gindele the disclosures of which are incorporated byreference herein.

It is known that gray level correction can be used to correct thebrightness and contrast of an image which is captured under anillumination condition where the subject is photographed alongside abright light source. Gamma correction and histogram correction aretypical examples of the gray level correction that can be used tocorrect such images. With gamma correction and histogram correction,however, because the image correction is performed using a fixedcoefficient, problems may arise where the image is clipped white due tooverexposure or in the case of under exposure, clipped black or obscuredby noise

It is known that adaptive gray level correction (adaptive enhancement)can be used to correct images, where the gray level values of pixelsadjacent to a pixel to be corrected are used to determine correctioncoefficients. With this approach, correction which adapts to the contentof an image can be achieved. An example of adaptive gray levelcorrection is disclosed in “Comparison of Retinex Models for HardwareImplementation” by Nosato et al., IEICE technical report, SIS, 2005-16,pp. 19-24 (June, 2005). This adaptive gray level correction is based onRetinex theory, which assumes that an input image is represented by aproduct of illumination light and reflectivity. Illumination light isseparated from an input image to thereby obtain a reflectivity image asa correction image. Given that an input image I is equal to anillumination light L times a reflectivity R (correction image), therelationship of R (x, y)=exp{log(I(x,y))−log(L(x,y))} can be achieved.Calculus of variation is used to estimate the illumination light, and aplurality of layers k with a resolution which is ½^(k) that of theoriginal image are generated. Calculations for updating the illuminationlight are repeated, starting from a layer with a lower resolution. Here,the calculation for updating the illumination light is performed usingthe expression L(x, y)=L(x, y)−μNSD X G(x, y), wherein G(x, y) is agradient of cost function and μNSD is a learning coefficient.Specifically, a processing, in which G(x, y) is first calculated, andμNSD (x, y) is then calculated, and based on these calculation results,L(x, y) is calculated, is repeated.

Further, JP2007-27967A discloses that, when a portrait photographingmode is selected by the photographer, an image is captured with theexposure value being set to a value less than the exposure valuenormally computed by an AE (Automatic Exposure) detector, and gray levelcorrection is applied to the image data from the image sensor by using agamma transform table for increasing the dynamic range of image datawhich has been subjected to gray level conversion processing, therebycorrecting the brightness value of portions of the image withinsufficient brightness which are located in the vicinity of the centerof the subject. Note that the portrait photographing mode must bemanually selected by the photographer, rather than being automaticallydetermined by the camera by analyzing preview image data.

SUMMARY

It is an object of the present disclosure to provide a digital camerathat uses preview image data to adjust the sensor exposure based onscene type, to improve the image quality of a final image.

This object is achieved by providing a digital camera, comprising: animage sensor for providing initial sensor image data and final sensorimage data; a lens for exposing the image of a scene onto the imagesensor; an exposure control system for adjusting an exposure level of afinal image on the image sensor in response to a scene type; and aprocessor for processing the initial sensor image data to select one ofa plurality of scene types, providing the selected scene type to theexposure control system prior to capturing the final sensor image datato cause the exposure control system to adjust the exposure level of thefinal image on the image sensor, wherein the exposure control systemsets the exposure level of the final image differently for differentscene types, and processing the final sensor image data in response tothe scene type to compensate for the exposure level of the final imageon the image sensor.

Advantages

It is an advantage of the present disclosure to provide a digital camerathat can produce high quality images for a range of different scenetypes.

It is an additional advantage of the present disclosure to provide adigital camera that can automatically determine a scene type and adjustan exposure level of a final image on an image sensor in response to thescene type.

It is a further advantage of the present disclosure to provide a digitalcamera that can automatically determine and apply digital processing tofinal image data from the image sensor, in response to the scene type,in order to compensate for the exposure level of the final image on theimage sensor.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 depicts a block diagram of a digital photography system,including a digital camera 300;

FIG. 2 depicts an image-processing diagram employed by the processor 320of the digital camera 300;

FIG. 3 depicts a flow diagram showing an embodiment of the presentinvention;

FIG. 4 depicts an example of an image-processing diagram employed foradaptive tone scale processing block 453 in FIG. 2;

FIG. 5 is a table showing a representative list of scene types intowhich it is advantageous to automatically classify captured scenes usingthe present invention;

FIG. 6 is a table showing the relative exposure level settings fordifferent scene types, both for scenes determined to contain faces andfor scenes determined to not contain any faces;

FIG. 7 is a flowchart showing the signal flow in an alternativeembodiment of the present invention;

FIG. 8 is a flowchart showing an example of a process for scene typeclassification; and

FIG. 9 is a table chart showing a relationship between a brightnesshistogram and an exposure correction amount according to the embodimentof the present disclosure.

DETAILED DESCRIPTION

The present disclosure provides a digital camera that includes analgorithm that processes initial sensor image data and selects the typeof scene to be captured, prior to exposing the image sensor for thefinal capture. The initial sensor image data is also used to provide apreview image, in order to compose the final image to be captured. Theexposure of the scene onto the sensor is automatically set based on thetype of scene determined by the algorithm. Different sensor exposureoffsets are used for these different scene types. As a result, thesensor output signal level changes based on the scene type. Digitaladaptive tone scale processing then compensates for this exposureoffset. This enables the digital camera to capture higher quality imagesfor a range of different scene types.

The present invention couples the behavior of the exposure program andthe still image processing pipeline, to optimally render the entiredynamic range of each scene type. For example: scenes with averagelighting conditions are slightly underexposed to avoid the loss(clipping) of highlight information; bright scenes—including backlitscenes—use lower-than-normal sensor exposure levels in order to preservethe highlight information; and text/document scenes use a higher thannormal sensor exposure levels in order to account for thehigher-than-average reflectance found in documents. Digital imageprocessing algorithms then adjust the tone reproduction in order toprovide a more pleasing processed final image than would have beenpossible without this coupled behavior.

Example scene types, listed in FIG. 5, include:

-   -   General—Used for “average” scenes, also serves as the default        type.    -   High Contrast Scene—Underexposes a scene to capture highlight        detail and uses adaptive tone scaling to optimally render the        large dynamic range of the scene.    -   Low Contrast Scene—Expands the contrast to optimally render the        relatively low dynamic range of the scene.    -   (Blue) Sky Scene—Sensitivity to blue sky results in preservation        of saturated colors, including sky tint.    -   Night Scene—Aggressively preserves fidelity of ‘black’ regions,        including nighttime content.    -   Color Saturated Scene—Maintains contrast to preserve saturated        colors.    -   Text Scene—Uses a higher than normal sensor exposure level in        order to account for the higher-than-average reflectance found        in documents.

These examples are not limiting, and other scene types that haveparticular exposure and tone scale characteristics such as scenes withsubjects located beyond the distance for flash use, may be found usefulto detect and process using the present invention.

FIG. 1 depicts a block diagram of a digital photography system,including a digital camera 300. Preferably, the digital camera 300 is aportable battery operated device, small enough to be easily handheld bya user when capturing and reviewing images. The digital camera 300produces digital images that are stored using image memory 330. Thedigital camera 300 includes a zoom lens 304 having an adjustableaperture and adjustable shutter 306. The zoom lens 304 is controlled byzoom and focus motor drives 308. The zoom lens 304 focuses light from ascene (not shown) on an image sensor 314, for example, a single-chipcolor CCD or CMOS image sensor. The output of the image sensor inconverted to digital form by Analog Signal Processor (ASP) and A/Dconverter 316, and temporarily stored in DRAM buffer memory 318. Thedata stored in the DRAM buffer memory 318 is subsequently processed by aprocessor 320 using firmware programs stored in firmware memory 328.Alternatively, the processor 320 can be provided by custom circuitry(e.g. by one or more custom integrated circuits (ICs) designedspecifically for use in digital cameras), or by a combination ofprogrammable processor(s) and custom circuits. The processed images arethen stored using image memory 330. It is understood that the imagememory 330 can be a removable Flash memory card, internal Flash memorychips, magnetic memory, or optical memory. In alternative embodiments,the digital camera 300 can also capture motion video images. Inalternative embodiments, the digital camera 300 can include otherfunctions, including the functions of a digital music player (e.g. MP3player), a mobile telephone, or a programmable digital assistant (PDA).

The image sensor 314 is controlled by a timing generator 312. The imagesensor 314 can have, for example, 10.2 megapixels (3680 pixels by 2760pixels). To provide a color image, the image sensor is overlaid with acolor filter array, which provides an image sensor having an array ofpixels that include different colored pixels. The different color pixelscan be arranged in many different patterns. As one example, thedifferent color pixels can be arranged using the well-known Bayer colorfilter array, as described in commonly assigned U.S. Pat. No. 3,971,065,“Color Imaging Array” to Bayer, the disclosure of which is incorporatedherein by reference. As a second example, the different color pixels canbe arranged as described in commonly assigned U.S. Publication No.US2005191729, filed on Jul. 28, 2007 and titled “Image Sensor withImproved Light Sensitivity” to Compton and Hamilton, the disclosure ofwhich is incorporated herein by reference. These examples are notlimiting, and many other color patterns may be used.

The image sensor 314 is effective when actuated in a first mode bytiming generator 312 for providing initial sensor image data, which isalso used for previewing the final image that will be captured in asecond mode. This preview mode sensor image data is provided as a lowresolution output, which can have, for example, 460 pixels by 345pixels, which is a factor of 8 fewer rows and columns of pixels than isnormally used for reading out the final image from the 10.2 megapixelsensor. The preview mode sensor image data can be provided by combiningvalues of adjacent pixels having the same color, or by eliminating someof the pixels values, or by combining some color pixels values whileeliminating other color pixel values. The preview mode image data can beprocessed as described in commonly assigned U.S. Pat. No. 6,292,218“Electronic camera for initiating capture of still images whilepreviewing motion images” which is incorporated herein by reference. Inan alternative embodiment, the digital camera 300 uses an opticalviewfinder, and the initial sensor image data is used only to determinethe exposure level of the final image, not to provide a preview images.

The image sensor 314 is also effective when actuated in a second mode bytiming generator 312 for providing final sensor image data. This finalmode sensor image data is provided as high resolution output image data,which for scenes having a high illumination level includes all of thepixels of the image sensor, and can be, for example, 10.2 megapixelfinal image data having 3680 pixels by 2760 pixels. At lowerillumination levels, the final sensor image data can be provided by“binning” some number of like-colored pixels on the image sensor, inorder to increase the signal level and thus the “ISO speed” of thesensor.

The zoom and focus motors 308 and the timing generator 312 arecontrolled by control signals supplied by processor 320. An exposurecontrol block 310 controls the exposure level of the image sensor 314,by controlling the f/number and exposure time of the adjustable apertureand adjustable shutter 306, and the gain (i.e. ISO speed) setting of theASP & A/D converter 316. The exposure control block 310 also controls aflash 302 which can illuminate the scene. While shown as a separateblock 310, it is understood that some, or all, of the functions of theexposure control block 310 can alternatively be performed by theprocessor 320.

The processor 320 also creates a lower-resolution image that can bereviewed on the color LCD image display 332. The graphical userinterface displayed on the color LCD image display 332 is controlled byuser controls 334. The user controls 334 are used to select variouscamera modes, such as the “preview image analysis” mode which will bedescribed later in reference to FIG. 3, as well as more traditionalmodes such as sports mode, night mode, landscape mode, and close-upmode. The user controls 334 are also used to turn on the camera, controlthe zoom lens, and initiate the picture taking process.

When the user sets the digital camera 300 in the “preview imageanalysis” mode and begins to take a picture, the processor 320automatically processes the preview sensor image data to select one of aplurality of scene types, to be described later in reference to FIG. 5.The processor 320 then communicates the selected scene type to theexposure control block 310 prior to initiating image capture of thefinal sensor image data. The exposure control block 310 then adjusts theexposure level of the final image on the image sensor, by settingappropriate flnumber, exposure time, and ASP gain values, such that theexposure level of the final image is set differently for the differentscene types. After the final sensor image data has been provided by theimage sensor 314, the processor 320 processes this final sensor imagedata, in response to the scene type, to compensate for the exposurelevel of the final image on the image sensor.

The processor 320 also provides additional calculations, such as focuscalculations used to focus the lens 304, during the preview mode. Theprocessor 320 also provides additional processing of the final sensorimage data, using the “standard image pipeline” depicted in FIG. 2, inorder to produce rendered sRGB image data which is stored within a“finished” image file, such as a well-known Exif-JPEG image file, in theimage memory 330.

The digital camera 300 can be connected via a dock interface 362 to adock/recharger 364, which is connected to a home computer 340. The dockinterface 362 can conform to, for example, the well-known USB 2.0interface specification. Alternatively, the interface between thedigital camera 300 and the home computer 340 can be a wirelessinterface, such as the well-known Bluetooth wireless interface or thewell-known 802.11 wireless interface. The home computer 340 can uploadimages via the Internet 370 to a photo service provider 372, such as theKodak EasyShare Gallery.

The processor 320 is coupled to a wireless modem 350, which enables thedigital camera 300 (which can also include mobile phone functions, asdescribed earlier) to transmit and receive information via an RF channel352. The wireless modem 350 communicates over a radio frequency (e.g.wireless) link with a mobile phone network 358, such as a 3GSM network.The mobile phone network 358 communicates with a photo service provider372, which can store digital images uploaded from the digital camera300. These images can be accessed via the Internet 370 by other devices,including the home computer 340.

FIG. 2 depicts an image-processing diagram employed by the processor 320of the digital camera 300 to effect the present invention. The previewsensor images 400 (which are also used to provide an electronicviewfinder on the color display 332 as the user composes the image to becaptured) provide input to the components of a preview image analysis410 image processing function. In a preferred embodiment, thisprocessing function is comprised of a set of distinct image analysiscapabilities consisting of, face detection 411, motion analysis 412,auto focus 413, auto white balance 414, and auto exposure 415. Suchdistinct image analysis capabilities are now standard in manycommercially available digital camera systems, and those skilled in thisart will be acquainted with their performance and the analysis resultsthey provide. These analysis results are used to perform scene typeclassification 416, resulting in the determination of the scene type 417for a given scene. The scene type classification 416 also provides arelative exposure level 419, which depends on the scene type, as will bedescribed later in reference to FIG. 6.

It is useful to further describe the motivation for the preview imageanalysis 410, to emphasize the novelty of this design. The exposurecontrol 310 sets the exposure of the scene onto the image sensor 314 inorder to best map the information in the scene to the dynamic range ofthe image sensor 314. The dynamic range is the ratio of the brightestdetectable object in a scene divided by the darkest detectable object.Any signals beyond the sensor saturation level, at the bright end of thedynamic range, are clipped by the image sensor 314, meaning that, beyondthe clipping point, adding more light does not increase the sensoroutput signal level. The dark end of the dynamic range is limited moregracefully. It gradually becomes enveloped by noise. The goal of previewimage analysis 410 is to detect the scene type 417 and select exposuresettings that balance between clipping of scene highlights and noisemasking of dark regions in the image. This method is an innovation overthe automatic exposure systems employed in prior art digital cameras,where the exposure settings are selected so the output image has thecorrect overall brightness, regardless of the dynamic range of thescene. Consideration of what image data is lost to clipping or noise, asa function of the scene dynamic range, is not part of these prior artcamera automatic exposure systems. As will be described in furtherdetail with respect to FIG. 3 and FIG. 4, the adaptive tone scaleprocessing 453 must work in concert with the scene exposure value 418and relative exposure level 419, to produce finished image file data 460with high image quality.

In addition to the determination of scene type 417, the outputs of thepreview image analysis function 410 also control other sub-systems ofthe digital camera 300 including the zoom & focus motor drivers 308 andthe exposure control 310. Changes in these sub-systems result in changesin the view finder images displayed on the color display 332. As shownin FIG. 2, the conventional scene exposure value 418 and the scene-typedependent relative exposure level 419 are used by an exposure program420 to determine appropriate exposure settings 425, including theexposure time and f/number of the adjustable aperture and adjustableshutter 306, and the gain (i.e., ISO speed) setting of the ASP & A/Dconverter 316, used during the capture of final sensor image data. Theexposure value that corresponds to the exposure setting 425 is equal tothe value of the conventional scene exposure value 418 shifted by anamount equal to the relative exposure level 419 for the particular scenetype 417 determined by scene type classification function 416.

The scene exposure settings 425 and the scene type 417 are used by arendering parameter program 430 to determine a set of renderingparameters used by a still image processing pipeline 450 toautomatically determine and apply digital processing to the final sensorimage data 440, in order to compensate for the exposure level of thefinal image on the image sensor. The rendering parameters 435 can alsoprovide scene type specific processing parameters for other processingfunctions, such as noise reduction and edge enhancement.

Those skilled in the art will be well acquainted with the basicprocessing blocks of the still image processing pipeline 450 in adigital camera 300. The standard image pipeline 452 consists ofoperations well known in the art, including de-mosaicing (also known ascolor pixel interpolation), noise reduction, white balance, edgeenhancement, and red eye reduction. Adaptive tone scale processing 453is then applied to final sensor image data 440, in concert with thestandard image pipeline 452, as part of the still image processingpipeline 450. The adaptive tone scale processing 453 will be describedin detail later, in reference to FIG. 4. Following these operations, theresulting processed image data arc compressed using standard compressionalgorithms (e.g. JPEG compression) and formatted to conform to standardimage formats, such as the well-known Exif-JPEG format, in block 454, inorder to produce finished image file data 460.

FIG. 3 depicts a flow diagram showing a first embodiment of the presentinvention. In block 500, a user sets the camera mode, using the usercontrols 334, to use the preview image analysis function 410 so that thedigital camera 300 will later determine the scene type 417 and theassociated exposure settings 425 and rendering parameters 435appropriate for that scene type 417. In block 510, the processor 320determines an approximate exposure to be used to capture the initialpreview sensor image data. In block 520, preview images arc captured andused to provide images for an electronic viewfinder on the color display332. These preview sensor images 400 are used by the preview imageanalysis processing function 410 to determine the scene type in block524.

The scene type can be determined as described in common assigned U.S.Publication No. US20070096024, titled “Image Capturing Apparatus” (whichclaims priority from JP 2007-121654 A), the disclosure of which isincorporated herein by reference.

After the user presses the shutter button (one of the user controls 334)in block 530, in block 526 the scene type 417 is used to set the sensorrelative exposure level 419. In other words, the exposure control systemsets the exposure level of the final image differently for the differentscene types shown in FIG. 6, since the relative exposure level 419depends on the type of scene detected. In block 540, the exposuresettings 425, which arc a function of the relative exposure level 419,are used to set the exposure conditions used to capture the final sensorimage data 440.

Following the setting of final exposure conditions in block 540, inblock 550 the final sensor image data 440 is captured. In preparationfor this capture, in block 528, a pixel binning mode is selected for lowlight scenes. The binning mode selected in block 528 is responsive tothe scene type, and is used in block 550 as the final sensor image data440 is captured. At high light levels, the binning mode is “no binning”,and all of the pixels of the image sensor 314 are output as final sensorimage data. As the light level is reduced to the “3 bin” threshold, thesignals from 3 pixels of the same color arc combined as the signal fromthe image sensor 314 is read out, in order to increase the signal level,while reducing the sensor resolution. As the light level is reducedfurther, to the “9 bin” threshold, the signals from 9 pixels of the samecolor are combined as the signal from the image sensor 314 is read out,in order to further increase the signal level, while reducing the sensorresolution. In the present invention, these “3 bin” and “9 bin”thresholds are set differently, for the different scene types (with andwithout faces detected) listed in FIG. 6

In block 529, rendering algorithm analysis is performed by the renderingparameter program 430 to determine the appropriate rendering parameters435 to use during digital compensation (e.g., adaptive tone scaleprocessing 453 and noise reduction and edge enhancement processing instandard image pipeline 452).

In block 560, the final sensor image data 440 is processed by thestandard image processing pipeline 452. This includes performingstandard image pipeline 452 functions (e.g., performing noise reduction,de-mosaicing, white balance, edge enhancement and red eye reduction)described earlier in reference to FIG. 2.

In block 570, the final sensor image data, having been processed by thestandard image processing pipeline 452, is processed using adaptive tonescale processing function 453, to adjust the contrast and to correct forthe exposure level of the final image on the image sensor, as will bedescribed later in reference to FIG. 4.

In block 580, the processed final image data is compressed and stored asfinished image file data 460 in the image memory 330.

FIG. 4 depicts an example of an image-processing diagram for adaptivetone scale processing 453 in FIG. 2. The input analysis image data 710,which is the output of the standard image processing pipeline 452 inFIG. 2, undergoes conversion to scene space encoding (e.g., ERIMM RGB asspecified in the ISO/TS 22028-3:2006 standard) in block 715. In block720, the rendering parameters 435 determined according to the selectedscene type 417 are retrieved and provided to component analysisoperations that include, scene balance analysis block 725, flarecorrection analysis block 730, and tone scale analysis block 735. Theresults of scene balance analysis block 725 and flare correctionanalysis block 730 are used in block 740 to perform color correctionprocessing (e.g. neutral balance, color balance, and flare correction)of the scene-space converted input analysis image data 710 as described,for example, in commonly-assigned U.S. Pat. No. 6,912,321 titled “Methodof Compensating a Digital Image for the Effects of Flare Light” toGindele, the disclosure of which is incorporated herein by reference. Inblock 760, the results of tone scale analysis block 735 are used toperform pedestal image tone scale application (e.g., contrastadjustment) as described, for example, in commonly-assigned U.S. Pat.No. 7,113,649, titled “Enhancing the Tonal Characteristics of DigitalImages” to Gindele, the disclosure of which is incorporated herein byreference.

A pedestal image is generated by performing luminance-chrominance (LCC)conversion on the color-corrected image data in block 745 andsubsequently splitting the luminance portion of the data, in luminancepedestal splitter block 750, into a pedestal image 753 and a textureimage 755. The tone scale of the pedestal image 753 us adjusted inpedestal image tone scale application block 760. The enhanced pedestalimage is recombined with the luminance texture image 755 in luminancepedestal recombiner block 765 to produce an enhanced luminance image.The enhanced luminance image and chrominance images 770 are collectivelycolor converted to three-color (e.g., RGB) data by the RGB conversion775 module. In block 780, the image data are converted to the outputcolor space appropriate for the finished image file data 460, such asthe sRGB output space.

FIG. 6 is a table showing the relative exposure level 419 for sevenspecific scene types, both for scenes determined to contain faces andfor scenes determined to not contain any faces. The relative exposurelevel values are given in exposure value (EV) units (i.e. photographicstops).

For high contrast scenes, the sensor is underexposed (relative to aphotographic normal exposure) by a one photographic stop (i.e. −1 EVunit) when a face is detected in the scene, and the relative exposurelevel is “4/3 EV when a face is not detected.

For low contrast scenes, the sensor is overexposed by +1/3 EV unit whena face is detected in the scene, and the relative exposure level is +2/3EV when a face is not detected.

For scenes that include significant patches of blue sky, the relativeexposure level is −2/3 EV when a face is detected in the scene, and −1EV when a face is not detected in the scene.

For night scenes, the relative exposure level is +2/3 EV when a face isdetected in the scene, and +1 EV when a face is not detected in thescene.

For scenes containing significant areas of high color saturation, therelative exposure level is −2/3 EV when a face is detected, and −1 EVwhen a face is not detected in the scene.

For documents and other type of text scenes, the relative exposure levelis +1 EV. If a face is detected in a scene, it is never classified as atext type scene. For all other scenes, the “general” category is used,and the relative exposure level is −1/3 EV when a face is detected, and−2/3 EV when a face is not detected in the scene.

As previously discussed, it is an important aspect of the presentinvention that the exposure control system be coupled to the adaptivetone scale processing 453. The example shown in FIG. 4 is not limitingand many other types of adaptive tone scale processing may be used. Inaddition, additional types of adaptive processing operations may beuseful to couple to exposure control settings as a function of selectedscene type 417 including, for example, color saturation, sharpnessadjustment, and noise reduction.

FIG. 7 is a flowchart showing the signal flow used in an alternativeembodiment of the present invention. A region dividing and averagingsection 12 divides preview image data 20 provided by the image sensor314 (see FIG. 1) into a plurality of regions or blocks, and calculatesand outputs an average pixel value for each region. For example, theregion dividing and averaging section 12 can divide the whole image datainto n x m rectangular regions (for example 12×8 regions) and calculatesa simple average of R, G, and B pixel values for each region.

An Exposure Assessment (EA) section 14 performs exposure assessment forpreview mode, based on the preview image data 20 to calculate anappropriate exposure value. The preview exposure assessment can beachieved either by measuring the light intensively in the center portionof the image, or using a multipoint light measurement method. Inaddition, the EA section 14 generates a brightness histogram using thepreview image 20, and provides the brightness histogram to an exposurecorrection amount calculation section 16.

More specifically, preview exposure assessment section 22 calculates anassessment value for controlling the exposure of the preview image. Theassessment value thus calculated is supplied to a preview exposurecontrol section 32 so that the preview image 20 is set to an appropriateexposure value. A photographing exposure prediction histogram generationsection 28, using the data supplied from the preview exposure assessmentsection 22, generates a brightness histogram. This histogram is whatwould be obtained from the image data of a still image which capturedwith the current exposure value. More specifically, the photographingexposure prediction histogram generation section 28 converts the averagevalue of the R, G, and B color values for each block (from regiondividing and averaging section 12) into brightness Y, and then convertsthe brightness Y into EV (exposure value) units equal to log 2(brightness Y/appropriate exposure value). After the EV value for eachblock is calculated, a histogram is generated. The horizontal axis ofthe histogram is the deviation amount (in EV units) relative to aso-called “normal” photographic exposure, and the vertical axis is thenumber of blocks with this deviation amount. The histogram calculated bythe exposure prediction histogram generation section 28 is provided tothe exposure correction amount calculation section 16.

The exposure correction amount calculation section 16 calculates anexposure correction amount with respect to the appropriate exposure, bydetermining whether or not the brightness histogram satisfies specificconditions, and supplies the exposure correction amount to the EAsection 14 and a gray level correction section 18. The exposurecorrection amount calculation section 16 calculates the exposurecorrection amount on the assumption that the image data of a capturedimage is to be subjected to gray level correction by the gray levelcorrection section 18.

The gray level correction section 18 applies adaptive gray levelcorrection to the image data of a still image captured with exposurecorrection, based on the exposure correction amount obtained from theexposure correction amount calculation section 16 and the appropriateexposure value (a reference of exposure). The gray level correctionsection 18 could use, for example, the adaptive tone scale processingdescribed earlier in relation to FIG. 4.

In alternative embodiments, gray level correction methods may be used toapplying a tone scale correction transformation to the brightness databy using an adaptive gamma function and by also correcting the colordifference data, thereby performing gray level correction. The graylevel correction section 18 performs gray level correction so thatexposure of the still image data captured with underexposure iscompensated for, to thereby achieve the appropriate exposure in thecorrected digital still image. The correction amount for the gray levelcorrection is decreased when the exposure correction amount is small andthe correction amount for the gray level correction is increased whenthe exposure correction amount is large. In the latter case, distortiondue to the gray level correction may be noticeable in some types ofimages.

FIG. 8 is a flowchart showing an example of a process for scene typeclassification according to some embodiments of the present invention.In this example, the exposure correction amount is set to one of sixlevels, relative to the appropriate exposure value for a “normal”exposure. More specifically, the exposure correction amount ΔEV is setto one of 0 EV, −Δ₁ EV, −Δ₂ EV, −Δ₃ EV, −Δ₄ EV, and +Δ₅ EV. Here, thepositive sign in front of the exposure correction amount indicatesoverexposure with respect to a “normal” exposure value, and the negativesign in front of the exposure correction amount indicates underexposurewith respect to a “normal” exposure value. Also, the relationship ofΔ₁<Δ₂<Δ₃<Δ₄ is satisfied.

First, in block 101 of FIG. 8, the processor 320 of the digital camera300 (see FIG. 1) determines whether or not the brightness level of thepreview image (if present) is lower than a level at which a strobe light(flash) needs to be on. Since the scene to be photographed is dark ifthe flash needs to be fired, this determination is made in order toprevent an extreme underexposure condition. Accordingly, if thebrightness level is lower than the flash ON level, the exposurecorrection amount ΔEV with respect to the approximate exposure value isset to 0 EV. Specifically, in this case, the appropriate “normal”exposure value is maintained (block 114).

If the brightness level of the preview image is equal to or higher thanthe level at which flash should be on, on the other hand, it is thendetermined whether or not the preview image data is not present (102).The preview image data does not exist immediately after the power to thedigital camera 300 is turned on, or immediately after the digital camera300 is switched from the image review mode to the image capture (e.g.photographing) mode. When a user presses the shutter button completelywith a single press when the camera is in review mode, exposurecorrection cannot be performed due to the absence of the preview imagedata. Accordingly, when there is no preview image data, higher priorityis given to photographing than to exposure correction, and the exposurecorrection amount calculation section 16 sets the exposure correctionamount to a default underexposure value (block 115). Specifically, theexposure correction amount calculation section 16 sets the exposurevalue to be less than the appropriate “normal” exposure value by anexposure correction amount equal to Δ₁ EV.

If the preview image is present, it is then determined whether or not abacklight condition is present in the scene to be photographed (block103). If a backlight condition is determined, the flash is fired toprovide “fill flash”. In this situation, the exposure correction amountcalculation section 16 sets the exposure correction amount to thedefault underexposure value of Δ₁ EV

If the scene does not have a blacklight, in block 104 it is determinedwhether or not the scene is a beach scene or a text (document). Thisdetermination is made because these types of scenes are preferably withcaptured using an exposure level greater than the “normal” exposurevalues, because of the high reflectance of the sand or paper in suchscenes. For these types of scenes, the exposure correction amountcalculation section 16 sets the exposure correction amount ΔEV to be +Δ₅EV (block 119), which is greater than the “normal” exposure value.Specifically, the exposure correction amount calculation section 16 setsthe exposure value to be over the appropriate exposure value by theexposure correction amount. The determination as to whether or not thescene is a beach scene or a text (document) photographing scene can beperformed by using well-known methods, such as common assigned US patentapplication US20070096024, “Image Capturing Apparatus”, the disclosureof which is incorporated herein by reference, which describes methodsfor determining the type of scene based on a combination of thetemperature, movement, hue, chroma, brightness, and other imagecharacteristics.

In block 105 of FIG. 8, if the photographic scene is not a specificscene type, it is then determined whether or not the ISO exposure index(provided by the image sensor 314 and ASP/A/D 316) is equal to orgreater than a threshold value. When the scene to be photographed isdark, the shutter speed is fixed so as to prevent the camera fromshaking and the ISO is increased to increase the sensitivity. However,this high ISO setting increases the image noise. Therefore, when the ISOis equal to or greater than a threshold value, the exposure correctionamount calculation section 16 sets the exposure correction amount ΔEV toequal 0 EV, so that a “normal” exposure level is maintained.

The ISO exposure index may be high either because: case (1) the scene isdark; or case (2) the scene is not dark, but a high-speed shutter (i.e.short exposure time) is used to freeze the motion of a moving subject,so the gain of the ASP & A/D converter 316 has been increased. In block106, the scene brightness level is checked to see if it's lower than themoving subject brightness. “Yes” corresponds to case (1), and theexposure correction amount ΔEV to set equal 0 EV (block 114), asdescribed above, to prevent a further increase in noise. “No”corresponds to case (2), where in block 107, a brightness difference isdetermined by using the histogram described earlier in reference to FIG.7. Then, in blocks 108-113, various conditions are checked to determinethe ΔEV value to be used to capture the final still image.

Specifically, in block 108, it is determined whether or not the sceneincludes a person's face. Methods for detecting a person's face are wellknown, and can include, for example, extracting regions of image datawhich include a skin color, and determining how well these regions matcha face template. If it is determined that the scene includes a person'sface in block 108, in block 109 the brightness histogram is checked todetermine whether or not it satisfies the conditions corresponding tothe exposure correction amount ΔEV=−Δ₃ EV. The condition for settingΔEV=−Δ₃ EV can be, for example, whether the histogram includes values(deviation amounts) of x4 EV or greater in 20% or more of the regions,no values of x5 EV or greater in any of the regions, values of x2 EV orless in 10% or less of the regions, and a face area with y2 EV orgreater and y3 EV or less.

If this condition for −Δ₃ EV is not satisfied in block 109, then inblock 110 it is determined whether or not the brightness histogramsatisfies the conditions corresponding to the exposure correction amountΔEV=−Δ₂ EV which can be, for example, whether the histogram includesvalues of x3 EV or greater in 60% or more of the regions, and a facearea with y1 EV or greater.

If the brightness histogram satisfies neither the −Δ₃ EV nor the −Δ₂ EVconditions, the exposure correction amount calculation section 16 setsthe exposure correction amount ΔEV to a default value of −Δ₁ EV (block115). If the brightness histogram satisfies the conditions for −Δ₂ EV,the exposure correction amount calculation section 16 sets the exposurecorrection amount ΔEV to a value of −Δ₂ EV (block 116). This means thatthe still image will be captured with the exposure value set to a valuewhich is lower than the default underexposure value. If the brightnesshistogram satisfies the conditions for −Δ₃ EV, the exposure correctionamount calculation section 16 sets the exposure correction amount ΔEV toa value of −Δ₁ EV (block 117). This means that the still image will becaptured with the exposure value being set to a value which is evenlower than the −Δ₃ EV underexposure value.

If the scene does not include a person's face, in block 111 it isdetermined whether or not the brightness histogram satisfies theconditions corresponding to −Δ₄ EV, and if not whether or not thebrightness histogram satisfies the conditions corresponding to −Δ₃ EV inblock 112, and if not whether or not the brightness histogram satisfiesthe conditions corresponding to −Δ₂ EV in block 113. The conditions forsetting the above exposure correction amounts can be, for example:

The conditions for −Δ₄ EV (without a face) are that the histogramincludes values of at least x4 EV in at least 20% of the regions, valuesof x5 EV or greater in at least one region, and values of x1 or less innot more than 10% of the regions, or values of x0 EV or less in 20% ormore of the regions.

The conditions for −Δ₃ EV (without a face) are that the histogramincludes values of x4 EV or greater in at least 25% of the regions, novalues of x5 EV or greater in any region, and values of x2 EV or smallerin not more than 10% of the regions.

The conditions for −Δ₂ EV (without a face) are that the histogramincludes values of x3 EV or greater in at least 50% of the regions.

All the conditions described above are shown in FIG. 9. Thedetermination criteria vary depending on the presence or absence of aperson's face in the scene, as shown by the columns in FIG. 9, in orderto make the correction amount in the gray level correction relativelysmaller when a person's face is included than when a person's face isnot included, to minimize noise in images that include faces.

The exposure correction amount ΔEV, which is calculated in the exposurecorrection amount calculation section 16, is used to set the finalexposure conditions when the still image is captured. The exposurecorrection amount ΔEV is also supplied to the gray level correctionsection 18, which provides adaptive tone scale processing in order tocompensate for the particular exposure correction amount used to capturethe still image. In other words, the exposure correction amount ΔEV isused both to adjust the image sensor exposure level for the final stillimage, and to perform digital adaptive tone correction on the capturedstill image data. Further, as shown in FIG. 9, the exposure correctionamount ΔEV is set smaller when a person's face is included in aphotographic scene than when a person's face is not included.

The invention has been described in detail with particular reference tocertain preferred embodiments thereof but it will be understood thatvariations and modifications can be effected within the spirit and scopeof the invention.

PARTS LIST

-   -   12 Region dividing and average section    -   14 Exposure assessment section    -   16 Exposure correction amount calculation section    -   18 Gray level correction section    -   20 Preview image    -   22 Preview exposure assessment section    -   26 Still image photographing exposure determination section    -   28 Photographing exposure prediction histogram generation        section    -   30 Exposure correction amount determination section    -   32 Preview control section    -   101 Block    -   102 Block    -   103 Block    -   104 Block    -   105 Block    -   106 Block    -   107 Block    -   108 Block    -   109 Block    -   110 Block    -   111 Block    -   112 Block    -   113 Block    -   114 Block    -   115 Block    -   116 Block    -   117 Block    -   118 Block    -   119 Block    -   330 digital camera    -   302 flash    -   304 zoom lens    -   306 adjustable aperture and adjustable shutter    -   308 zoom and focus motor drives    -   310 exposure control block    -   312 timing generator    -   314 image sensor    -   316 ASP and A/D converter    -   318 DRAM buffer memory    -   320 processor    -   328 firmware memory    -   330 image memory    -   332 color LCD image display    -   334 user controls    -   340 home computer    -   350 wireless modem    -   352 RF channel    -   358 mobile phone network    -   362 dock interface    -   364 dock/recharger    -   370 Internet    -   372 photo service provider    -   400 view finder images    -   410 preview image analysis    -   411 face detection    -   412 motion analysis    -   413 auto focus    -   414 auto white balance    -   415 auto exposure    -   416 scene type classification    -   417 scene type    -   418 scene exposure value    -   419 relative exposure level    -   420 exposure program    -   425 exposure settings    -   430 rendering parameter program    -   435 rendering parameters    -   440 final sensor image data    -   450 still image processing pipeline    -   452 standard image pipeline    -   453 adaptive tone scale processing    -   454 file finishing and compression    -   460 finished image file data    -   500 block    -   510 block    -   520 block    -   524 block    -   526 block    -   528 block    -   529 block    -   530 block    -   540 block    -   550 block    -   560 block    -   570 block    -   580 block    -   710 input analysis image data    -   715 conversion to scene space    -   720 retrieve parameters for scene type    -   725 scene balance analysis    -   730 flare correction analysis    -   735 tone scale analysis    -   740 color correction processing    -   745 LCC conversion    -   750 luminance pedestal splitter    -   753 pedestal image    -   755 texture image    -   760 pedestal image tone scale application    -   765 luminance pedestal recombiner    -   770 chrominance images    -   775 RGB conversion    -   780 conversion to output color space

What is claimed is:
 1. An image processing device comprising: an imagesensor configured to provide initial sensor image data and final sensorimage data; a lens configured to expose an image of a scene onto theimage sensor; a processor configured to: process the initial sensorimage data to select one of a plurality of pre-determined scene types,wherein the pre-determined scene types include at least one scene typeselected from the group including high-contrast scene, low-contrastscene, sky scene, night scene, color-saturated scene, and text/documentscene; and cause an exposure control system to adjust the exposure levelfor a final image according to the selected scene type; and the exposurecontrol system configured to adjust the exposure level for capturing thefinal image on the image sensor by setting, according to signals fromthe processor, the exposure level for the final image to be a defaultscene exposure level changed by a pre-determined relative exposure levelcorresponding to the selected scene type, where the relative exposurelevel change is an increase in exposure if the scene type islow-contrast scene, night scene, or text/document scene, and a decreasein exposure if the scene type is high-contrast scene, sky scene, orcolor-saturated scene.
 2. The device of claim 1, wherein the processoris configured to use the initial sensor image data to provide a previewimage for display on a color display.
 3. The device of claim 1, whereinthe processor is configured to use the initial sensor image data todetect faces, and wherein processing the initial image sensor data toselect one of a plurality of pre-determined scene types is based in parton whether or not a face is detected in the initial image data.
 4. Thedevice of claim 1, wherein selecting the scene type further comprisesusing an assessment of the light intensity in the center portion of theinitial sensor image data to select the scene type.
 5. The device ofclaim 1, wherein the initial sensor image data is lower resolution thanthe final sensor image data.
 6. The device of claim 1, wherein settingthe exposure level for the final image to be the default scene exposurelevel changed by the pre-determined relative exposure level causes thedynamic range of the image sensor to accommodate the range of brightnessin the scene.
 7. The device of claim 1, wherein the processor isconfigured to apply adaptive tone scale processing to the final imagesensor data by using a set of pre-determined adaptive tone scaleprocessing parameters corresponding to the selected scene type.
 8. Thedevice of claim 1, wherein the processor is configured to process thefinal sensor image data by using a pre-determined pipeline processingparameter corresponding to the selected scene type, which is selectedfrom a group of parameter types including: neutral balance, colorbalance, flare correction, contrast adjustment, color saturation,sharpness adjustment, noise reduction, and combinations thereof.
 9. Amethod for using initial sensor image data to improve final sensor imagedata, comprising: by a processor of an image processing device,processing initial sensor image data provided by an image sensor toselect one of a plurality of pre-determined scene types, wherein thepre-determined scene types include at least one scene type selected fromthe group including high-contrast scene, low-contrast scene, sky scene,night scene, color-saturated scene, and text/document scene; causing, bythe processor, an exposure control system to adjust the exposure levelfor a final image according to the selected scene type; and adjusting,by an exposure control system, the exposure level for capturing a finalimage on the image sensor by setting, according to signals from theprocessor, the exposure level for the final image to be a default sceneexposure level changed by a pre-determined relative exposure levelcorresponding to the selected scene type, where the relative exposurelevel change is an increase in exposure if the scene type islow-contrast scene, night scene, or text/document scene, and a decreasein exposure if the scene type is high-contrast scene, sky scene, orcolor-saturated scene.
 10. An image processing device comprising: animage sensor configured to provide initial sensor image data and finalsensor image data; a lens configured to expose an image of a scene ontothe image sensor; a processor configured to: generate an array ofaverage values for a plurality of n×m regions based on the initialsensor image data; use the array of average values to select one of aplurality of pre-determined scene types; and cause an exposure controlsystem to adjust the exposure level for a final image according to theselected scene type; and the exposure control system configured toadjust the exposure level for capturing the final image on the imagesensor by setting, according to signals from the processor, the exposurelevel for the final image to be a default scene exposure level changedby a pre-determined relative exposure level corresponding to theselected scene type.
 11. The device of claim 10, wherein the processoris configured to use the initial sensor image data to provide a previewimage for display on a color display.
 12. The device of claim 10,wherein the processor is configured to use the initial image data todetect faces, and wherein selecting one of a plurality of predeterminedscene types is further based in part on whether or not a face isdetected in the initial image data.
 13. The device of claim 10, whereinthe array of average values is converted to an array of brightnessvalues, and the array of brightness values is used to select the scenetype.
 14. The device of claim 10, wherein selecting the scene typefurther comprises using an assessment of the light intensity in thecenter portion of the initial sensor image data to select the scenetype.
 15. The device of claim 10, wherein generating an array of averagevalues comprises dividing the initial sensor image data into a pluralityof regions.
 16. The device of claim 10, wherein generating an array ofaverage values comprises calculating an average of R, G, and B pixelvalues for each region.
 17. The device of claim 10, wherein the initialsensor image data is lower resolution than the final sensor image data.18. The device of claim 10, wherein setting the exposure level for thefinal image to be the default scene exposure level changed by thepre-determined relative exposure level causes the dynamic range of theimage sensor to accommodate the range of brightness in the scene. 19.The device of claim 10, wherein the value of n may equal the value of mfor the plurality of n×m regions.
 20. The device of claim 10, whereinthe processor is configured to process the final sensor image data byusing a pre-determined pipeline processing parameter corresponding tothe selected scene type, which is selected from a group of parametertypes including: neutral balance, color balance, flare correction,contrast adjustment, color saturation, sharpness adjustment, noisereduction, and combinations thereof.